Sequential Feature Extraction Using Information-Theoretic Learning

A classification system typically includes both a feature extractor and a classifier. The two components can be trained either sequentially or simultaneously. The former option has an implementation advantage since the extractor is trained independently of the classifier, but it is hindered by the suboptimality of feature selection. Simultaneous training has the advantage of minimizing classification error, but it has implementation difficulties. Certain criteria, such as Minimum Classification Error, are better suited for simultaneous training, while other criteria, such as Mutual Information, are amenable for training the extractor either sequentially or simultaneously. Herein, an information-theoretic criterion is introduced and is evaluated for sequential training, in order to ascertain its ability to find relevant features for classification. The proposed method uses non-parametric estimation of Renyi’s entropy to train the extractor by maximizing an approximation of the mutual information between the class labels and the output of the extractor. The proposed method is compared against seven other feature reduction methods and, when combined with a simple classifier, against the Support Vector Machine and Optimal Hyperplane. Interestingly, the evaluations show that the proposed method, when used in a sequential manner, performs at least as well as the best simultaneous feature reduction methods. Index Terms -Feature extraction, Information theory, Classification, Nonparametric statistics. study. Another method, Vapnik’s Structural Risk Minimization (and its embodiment, the Support Vector Machine (SVM) [4]-[10]), has recently been determined to provide more explicit control of generalization through regularization. As such, it has become the obvious methodology with which to compare the performance of any feature reduction method. Feature reduction methods may be categorized based on whether the projector and the classifier are trained sequentially or simultaneously. Sequential methods adapt the projector based on optimizing a criterion at the output of the projector. On the other hand, simultaneous methods adapt the projector based on optimizing a criterion at the output of the classifier. The former is independent of the classifier, while the latter is trained “through” the classifier. This gives sequential methods an implementation advantage. Not only does it take longer to train simultaneously due to the increased computational complexity, but also the cost function landscapes may become more difficult to search. Moreover, a new set of update equations must be derived and the projector must be re-trained if it is desired to evaluate a new classifier. On the other hand, simultaneous methods have the obvious (theoretically speaking) performance advantage in that they optimize the projector and the classifier together, that is they tune the projector to the classifier discriminant functions. It is expected that, every thing else remaining constant, simultaneous training will produce superior results compared to training sequentially. Consequently, the onus is on sequential training methods to demonstrate that there exists an easily implementable and general-purpose feature extraction algorithm that provides, with the combination of a suitable classifier, commensurate classification error. Another difference between sequential and simultaneous systems is the choice of possible criteria for use in training the extractor. Criteria such as Minimum Classification Error (MCE) [1]-[3] and Mean Square Error (MSE) [11], for example, are well suited for training the extractor simultaneously. However, they are not appropriate for sequential training since both of these criteria are based on an error signal. In order to use the MSE criterion for sequential training, a set of NO-dimensional targets (one for each class) must be defined in the output feature space, yj(k). There is no principled method known to the authors for selecting these targets in the feature space, combined with the expectation that the classification performance will vary considerably depending on the choice of targets used. Similarly, the MCE criterion is based on the assumption that the relative values of the output of the (in this case) projector, are related to the a posteriori probabilities of each class, to wit, it is based on the assumption that the values are the output of a classifier. Other criteria, on the other hand, are well suited for either sequential or simultaneous training. For example, an information-theoretic method that makes use of mutual information (MI) could be used. In this case, the extractor can be trained either τ1(k) Sum of